[Security Solution] Migrates siem-detection-engine-rule-status alertId to saved object references array #114585

spong · 2021-10-12T01:17:02Z

Summary

Resolves (a portion of) #107068 for the siem-detection-engine-rule-status type by migrating the alertId to be within the SO references[]. Based on: #113577

Migrates the legacy siem-detection-engine-rule-status alertId to saved object references array
Adds an e2e test for siem-detection-engine-rule-status
Breaks out siem-detection-engine-rule-status & security-rule SO's to their own dedicated files/directories, and cleaned up typings/imports

Before migration you can observe the existing data structure of siem-detection-engine-rule-status via Dev tools as follows:

GET .kibana/_search
{
  "size": 10000, 
  "query": {
    "term": {
      "type": {
        "value": "siem-detection-engine-rule-status"
      }
    }
  }
}

{
  "_index" : ".kibana-spong_8.0.0_001",
  "_id" : "siem-detection-engine-rule-status:d580f1a0-2afe-11ec-8621-8d6bfcdfd75e",
  "_score" : 2.150102,
  "_source" : {
    "siem-detection-engine-rule-status" : {
      "alertId" : "d62d2980-27c4-11ec-92b0-f7b47106bb35", <-- alertId which we want in the references array and removed
      "statusDate" : "2021-10-12T01:50:52.898Z",
      "status" : "failed",
      "lastFailureAt" : "2021-10-12T01:50:52.898Z",
      "lastSuccessAt" : "2021-10-12T01:18:29.195Z",
      "lastFailureMessage" : "6 minutes (385585ms) were not queried between this rule execution and the last execution, so signals may have been missed. Consider increasing your look behind time or adding more Kibana instances. name: \"I am the Host who Names!\" id: \"d62d2980-27c4-11ec-92b0-f7b47106bb35\" rule id: \"214ccef6-e98e-493a-98c5-5bcc2d497b79\" signals index: \".siem-signals-spong-default\"",
      "lastSuccessMessage" : "succeeded",
      "gap" : "6 minutes",
      "lastLookBackDate" : "2021-10-07T23:43:27.961Z"
    },
    "type" : "siem-detection-engine-rule-status",
    "references" : [ ],
    "coreMigrationVersion" : "7.14.0",
    "updated_at" : "2021-10-12T01:50:53.404Z"
  }
}

Post migration the data structure should be updated as follows:

{
  "_index": ".kibana-spong_8.0.0_001",
  "_id": "siem-detection-engine-rule-status:d580f1a0-2afe-11ec-8621-8d6bfcdfd75e",
  "_score": 2.1865466,
  "_source": {
    "siem-detection-engine-rule-status": {
      "statusDate": "2021-10-12T01:50:52.898Z", <-- alertId is no more!
      "status": "failed",
      "lastFailureAt": "2021-10-12T01:50:52.898Z",
      "lastSuccessAt": "2021-10-12T01:18:29.195Z",
      "lastFailureMessage": "6 minutes (385585ms) were not queried between this rule execution and the last execution, so signals may have been missed. Consider increasing your look behind time or adding more Kibana instances. name: \"I am the Host who Names!\" id: \"d62d2980-27c4-11ec-92b0-f7b47106bb35\" rule id: \"214ccef6-e98e-493a-98c5-5bcc2d497b79\" signals index: \".siem-signals-spong-default\"",
      "lastSuccessMessage": "succeeded",
      "gap": "6 minutes",
      "lastLookBackDate": "2021-10-07T23:43:27.961Z"
    },
    "type": "siem-detection-engine-rule-status",
    "references": [
      {
        "id": "d62d2980-27c4-11ec-92b0-f7b47106bb35", <-- previous alertId has been converted to references[]
        "type": "alert",
        "name": "alert_0"
      }
    ],
    "migrationVersion": {
      "siem-detection-engine-rule-status": "7.16.0"
    },
    "coreMigrationVersion": "8.0.0",
    "updated_at": "2021-10-12T01:50:53.406Z"
  }
},

Manual testing

There are e2e tests but for any manual testing or verification you can do the following:

Manual upgrade test

If you have a 7.15.0 system and can migrate it forward that is the most straight forward way to ensure this does migrate correctly. You should see that the Rule Monitoring table and Rule Details Failure History table continue to function without error.

Downgrade via script and test migration on kibana reboot

If you have a migrated Rule Status SO and want to test the migration, you can run the below script to downgrade the status SO then restart Kibana and observe the migration on startup.

Note: Since this PR removes the mapping, you would need to update the SO mapping to include alertId again else you will receive a strict/dynamic mapping error.

# Replace id w/ correct Rule Status SO id of existing migrated object
POST .kibana/_update/siem-detection-engine-rule-status:d580ca91-2afe-11ec-8621-8d6bfcdfd75e
{
  "script" : {
    "source": """
    ctx._source.migrationVersion['siem-detection-engine-rule-status'] = "7.15.0";
    ctx._source['siem-detection-engine-rule-status'].alertId = ctx._source.references[0].id;
    ctx._source.references.remove(0);
    """,
    "lang": "painless"
  }
}

Restart Kibana and now it should be migrated correctly and you shouldn't see any errors in your console. You should also see that the Rule Monitoring table and Rule Details Failure History table continue to function without error.

Checklist

Delete any items that are not applicable to this PR.

~~Documentation was added for features that require explanation or tutorials~~
Unit or functional tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

…detection-engine-rule-status-references

…to references for create API

…detection-engine-rule-status-references

elasticmachine · 2021-10-14T02:13:56Z

Pinging @elastic/security-detections-response (Team:Detections and Resp)

elasticmachine · 2021-10-14T02:13:58Z

Pinging @elastic/security-solution (Team: SecuritySolution)

…detection-engine-rule-status-references

…s bugs

…detection-engine-rule-status-references

spong · 2021-10-14T22:12:56Z

...etection_engine/rule_execution_log/saved_objects_adapter/rule_status_saved_objects_client.ts

+            },
+            aggs: {
+              rule_status: {
+                reverse_nested: {},


Ended up running into issues going the two query route as well too as it wasn't possible to do a nested top_hits agg and sort on a parent field without specifying reverse_nested.... 😅

Touched base w/ @XavierM and just ended up adding support for reverse_nested here. So now we're still able to complete this operation in a single query, and can keep the same extractions logic below as before. 🙂

spong · 2021-10-14T22:19:57Z

...etection_engine/rule_execution_log/saved_objects_adapter/rule_status_saved_objects_client.ts

+  create: (
+    attributes: IRuleStatusSOAttributes,
+    options?: SavedObjectsCreateOptions
+  ) => Promise<SavedObject<IRuleStatusSOAttributes>>;
  update: (
    id: string,
-    attributes: Partial<IRuleStatusSOAttributes>
+    attributes: Partial<IRuleStatusSOAttributes>,
+    options?: SavedObjectsCreateOptions


Create/Update interfaces updated to include SavedObjectsCreateOptions so the SO references[] can be supplied and the correct document is updated (otherwise we'll end up with new docs w/o references[]).

…nt is always updated/created

spong · 2021-10-14T22:25:18Z

.../security_solution/server/lib/detection_engine/rules/legacy_rule_status/legacy_migrations.ts

+export const truncateMessageFields: SavedObjectMigrationFn<Record<string, unknown>> = (doc) => {
+  const { lastFailureMessage, lastSuccessMessage, ...restAttributes } = doc.attributes;
+
+  return {
+    ...doc,
+    attributes: {
+      lastFailureMessage: truncateMessage(lastFailureMessage),
+      lastSuccessMessage: truncateMessage(lastSuccessMessage),
+      ...restAttributes,
+    },
+    references: doc.references ?? [],
+  };
+};


From recently merged #112257 -- copied over as part of breaking out this and the security-rule SO mappings .

spong · 2021-10-14T22:30:30Z

...er/lib/detection_engine/rules/legacy_rule_status/legacy_rule_status_saved_object_mappings.ts

+export const ruleStatusSavedObjectMappings: SavedObjectsType['mappings'] = {
+  properties: {
+    status: {
+      type: 'keyword',
+    },


I ended up removing the alertId mapping. Spoke with @FrankHassanabad about this and he ended up leaving the mapping in some instances and just removed the field from the usable types. We should be fine with it removed here (and the types), but something to note when testing. Also means you can't use the painless script to downgrade and test as there will be a mapping mis-match.

spong · 2021-10-14T22:41:20Z

x-pack/plugins/security_solution/server/lib/detection_engine/rules/types.ts

@@ -111,7 +111,6 @@ export type RuleAlertType = SanitizedAlert<RuleParams>;

 // eslint-disable-next-line @typescript-eslint/no-explicit-any
 export interface IRuleStatusSOAttributes extends Record<string, any> {


I attempted to remove extends Record<string, any> as this is polluting this interface downstream and preventing type errors from invalid fields (i.e. alertId), but ran into issues trying to resolve the types within mergeStatuses and other utilities. Would be happy to pair and try to clean this up, but we're close to removing this SO and related utilities, so not a must unless time permits.

kibana/x-pack/plugins/security_solution/server/lib/detection_engine/routes/utils.ts

Line 208 in de43a3b

export const mergeStatuses = (

...etection_engine/rule_execution_log/saved_objects_adapter/rule_status_saved_objects_client.ts

pgayvallet

LGTM for core changes

...etection_engine/rule_execution_log/saved_objects_adapter/rule_status_saved_objects_client.ts

x-pack/plugins/security_solution/server/lib/detection_engine/routes/utils.test.ts

…references

banderror · 2021-10-18T09:56:10Z

@elasticmachine merge upstream

…references

banderror · 2021-10-18T09:58:10Z

Resolved a merge conflict ⬆️ in x-pack/test/detection_engine_api_integration/security_and_spaces/tests/migrations.ts

banderror

It looks really good to me, thank you for figuring out the problem with nested/reverse_nested aggregations and for making the code cleaner 👍

I noticed what I believe is a bug, I'll try to quickly push a fix in this branch if I can.

banderror · 2021-10-18T10:33:22Z

.../security_solution/server/lib/detection_engine/rules/legacy_rule_status/legacy_migrations.ts

+    return {
+      ...doc,
+      attributes: {
+        ...attributesWithoutAlertId.attributes,
+      },


Hmmm, I think it should be attributes: attributesWithoutAlertId. attributesWithoutAlertId.attributes will be undefined.

Nice catch -- thanks @banderror! Looks like this was a ©️🍝 from Frank's implementation. Here's the follow-up PR they did which includes the fix/test: https://github.com/elastic/kibana/pull/115098/files#diff-9729eb10e272059f7d9571cb208769b3e93fb73a5ac8483128d16dfc577e7057L248.

banderror · 2021-10-18T10:34:17Z

.../security_solution/server/lib/detection_engine/rules/legacy_rule_status/legacy_migrations.ts

+    // early return if alertId is not a string as expected
+    return { ...doc, references: existingReferences };


Nit: return doc;? :)

I think we still want this otherwise references won't be an empty array since it's absent in the original doc, no?

I'm not sure if the platform treats empty array references and undefined the same, so would test before making the change.

banderror · 2021-10-18T10:48:26Z

x-pack/test/detection_engine_api_integration/security_and_spaces/tests/migrations.ts

@@ -86,6 +86,33 @@ export default ({ getService }: FtrProviderContext): void => {
          '7d'
        );
      });
+
+      it('migrates legacy siem-detection-engine-rule-status to use saved object references', async () => {


Another test checking that all the other attributes are migrated successfully would help to catch the bug in the migration function.

There's a similar one for siem-detection-engine-rule-actions:

it('migrates legacy siem-detection-engine-rule-actions and retains "ruleThrottle" and "alertThrottle" as the same attributes as before', async () => {

banderror

So there's an upcoming Jenkins upgrade in less than 2 hours:

The update process does cause downtime, stopping new builds from running and waiting as necessary for running jobs to complete before updating.

I'd like to merge this PR as soon as CI is 🟢 -- really need these changes in master for my own branch.

I'll submit the fix and a test for it in a separate PR.

Thanks again 🙏

kibanamachine · 2021-10-18T12:19:31Z

💚 Build Succeeded

Metrics [docs]

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/development-plugin-saved-objects.html#_mappings

id	before	after	diff
`siem-detection-engine-rule-status`	12	11	-1

History

💚 Build #161209 succeeded ddccdfe
💚 Build #161072 succeeded 2bf2bd9
💔 Build #160760 failed 122f015
💔 Build #160267 failed bd30df7
💔 Build #159798 failed bec9779

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @spong

banderror · 2021-10-18T12:53:50Z

Auto-backport didn't work. I guess it's because I was not the author of the PR and merged it? 🤔
Manual backport here: #115359

banderror · 2021-10-18T12:54:11Z

Follow-up fix here: #115355

…d to saved object references array (elastic#114585) ## Summary Resolves (a portion of) elastic#107068 for the `siem-detection-engine-rule-status` type by migrating the `alertId` to be within the `SO references[]`. Based on: elastic#113577 * Migrates the legacy `siem-detection-engine-rule-status` `alertId` to saved object references array * Adds an e2e test for `siem-detection-engine-rule-status` * Breaks out `siem-detection-engine-rule-status` & `security-rule` SO's to their own dedicated files/directories, and cleaned up typings/imports Before migration you can observe the existing data structure of `siem-detection-engine-rule-status` via Dev tools as follows: ``` GET .kibana/_search { "size": 10000, "query": { "term": { "type": { "value": "siem-detection-engine-rule-status" } } } } ``` ``` JSON { "_index" : ".kibana-spong_8.0.0_001", "_id" : "siem-detection-engine-rule-status:d580f1a0-2afe-11ec-8621-8d6bfcdfd75e", "_score" : 2.150102, "_source" : { "siem-detection-engine-rule-status" : { "alertId" : "d62d2980-27c4-11ec-92b0-f7b47106bb35", <-- alertId which we want in the references array and removed "statusDate" : "2021-10-12T01:50:52.898Z", "status" : "failed", "lastFailureAt" : "2021-10-12T01:50:52.898Z", "lastSuccessAt" : "2021-10-12T01:18:29.195Z", "lastFailureMessage" : "6 minutes (385585ms) were not queried between this rule execution and the last execution, so signals may have been missed. Consider increasing your look behind time or adding more Kibana instances. name: \"I am the Host who Names!\" id: \"d62d2980-27c4-11ec-92b0-f7b47106bb35\" rule id: \"214ccef6-e98e-493a-98c5-5bcc2d497b79\" signals index: \".siem-signals-spong-default\"", "lastSuccessMessage" : "succeeded", "gap" : "6 minutes", "lastLookBackDate" : "2021-10-07T23:43:27.961Z" }, "type" : "siem-detection-engine-rule-status", "references" : [ ], "coreMigrationVersion" : "7.14.0", "updated_at" : "2021-10-12T01:50:53.404Z" } } ``` Post migration the data structure should be updated as follows: ``` JSON { "_index": ".kibana-spong_8.0.0_001", "_id": "siem-detection-engine-rule-status:d580f1a0-2afe-11ec-8621-8d6bfcdfd75e", "_score": 2.1865466, "_source": { "siem-detection-engine-rule-status": { "statusDate": "2021-10-12T01:50:52.898Z", <-- alertId is no more! "status": "failed", "lastFailureAt": "2021-10-12T01:50:52.898Z", "lastSuccessAt": "2021-10-12T01:18:29.195Z", "lastFailureMessage": "6 minutes (385585ms) were not queried between this rule execution and the last execution, so signals may have been missed. Consider increasing your look behind time or adding more Kibana instances. name: \"I am the Host who Names!\" id: \"d62d2980-27c4-11ec-92b0-f7b47106bb35\" rule id: \"214ccef6-e98e-493a-98c5-5bcc2d497b79\" signals index: \".siem-signals-spong-default\"", "lastSuccessMessage": "succeeded", "gap": "6 minutes", "lastLookBackDate": "2021-10-07T23:43:27.961Z" }, "type": "siem-detection-engine-rule-status", "references": [ { "id": "d62d2980-27c4-11ec-92b0-f7b47106bb35", <-- previous alertId has been converted to references[] "type": "alert", "name": "alert_0" } ], "migrationVersion": { "siem-detection-engine-rule-status": "7.16.0" }, "coreMigrationVersion": "8.0.0", "updated_at": "2021-10-12T01:50:53.406Z" } }, ``` #### Manual testing --- There are e2e tests but for any manual testing or verification you can do the following: ##### Manual upgrade test If you have a 7.15.0 system and can migrate it forward that is the most straight forward way to ensure this does migrate correctly. You should see that the `Rule Monitoring` table and Rule Details `Failure History` table continue to function without error. ##### Downgrade via script and test migration on kibana reboot If you have a migrated `Rule Status SO` and want to test the migration, you can run the below script to downgrade the status SO then restart Kibana and observe the migration on startup. Note: Since this PR removes the mapping, you would need to [update the SO mapping](https://github.com/elastic/kibana/pull/114585/files#r729386126) to include `alertId` again else you will receive a strict/dynamic mapping error. ```json # Replace id w/ correct Rule Status SO id of existing migrated object POST .kibana/_update/siem-detection-engine-rule-status:d580ca91-2afe-11ec-8621-8d6bfcdfd75e { "script" : { "source": """ ctx._source.migrationVersion['siem-detection-engine-rule-status'] = "7.15.0"; ctx._source['siem-detection-engine-rule-status'].alertId = ctx._source.references[0].id; ctx._source.references.remove(0); """, "lang": "painless" } } ``` Restart Kibana and now it should be migrated correctly and you shouldn't see any errors in your console. You should also see that the `Rule Monitoring` table and Rule Details `Failure History` table continue to function without error. ### Checklist Delete any items that are not applicable to this PR. - [ ] ~[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials~ - [X] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### For maintainers - [x] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) Co-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

…d to saved object references array (#114585) (#115359) ## Summary Resolves (a portion of) #107068 for the `siem-detection-engine-rule-status` type by migrating the `alertId` to be within the `SO references[]`. Based on: #113577 * Migrates the legacy `siem-detection-engine-rule-status` `alertId` to saved object references array * Adds an e2e test for `siem-detection-engine-rule-status` * Breaks out `siem-detection-engine-rule-status` & `security-rule` SO's to their own dedicated files/directories, and cleaned up typings/imports Before migration you can observe the existing data structure of `siem-detection-engine-rule-status` via Dev tools as follows: ``` GET .kibana/_search { "size": 10000, "query": { "term": { "type": { "value": "siem-detection-engine-rule-status" } } } } ``` ``` JSON { "_index" : ".kibana-spong_8.0.0_001", "_id" : "siem-detection-engine-rule-status:d580f1a0-2afe-11ec-8621-8d6bfcdfd75e", "_score" : 2.150102, "_source" : { "siem-detection-engine-rule-status" : { "alertId" : "d62d2980-27c4-11ec-92b0-f7b47106bb35", <-- alertId which we want in the references array and removed "statusDate" : "2021-10-12T01:50:52.898Z", "status" : "failed", "lastFailureAt" : "2021-10-12T01:50:52.898Z", "lastSuccessAt" : "2021-10-12T01:18:29.195Z", "lastFailureMessage" : "6 minutes (385585ms) were not queried between this rule execution and the last execution, so signals may have been missed. Consider increasing your look behind time or adding more Kibana instances. name: \"I am the Host who Names!\" id: \"d62d2980-27c4-11ec-92b0-f7b47106bb35\" rule id: \"214ccef6-e98e-493a-98c5-5bcc2d497b79\" signals index: \".siem-signals-spong-default\"", "lastSuccessMessage" : "succeeded", "gap" : "6 minutes", "lastLookBackDate" : "2021-10-07T23:43:27.961Z" }, "type" : "siem-detection-engine-rule-status", "references" : [ ], "coreMigrationVersion" : "7.14.0", "updated_at" : "2021-10-12T01:50:53.404Z" } } ``` Post migration the data structure should be updated as follows: ``` JSON { "_index": ".kibana-spong_8.0.0_001", "_id": "siem-detection-engine-rule-status:d580f1a0-2afe-11ec-8621-8d6bfcdfd75e", "_score": 2.1865466, "_source": { "siem-detection-engine-rule-status": { "statusDate": "2021-10-12T01:50:52.898Z", <-- alertId is no more! "status": "failed", "lastFailureAt": "2021-10-12T01:50:52.898Z", "lastSuccessAt": "2021-10-12T01:18:29.195Z", "lastFailureMessage": "6 minutes (385585ms) were not queried between this rule execution and the last execution, so signals may have been missed. Consider increasing your look behind time or adding more Kibana instances. name: \"I am the Host who Names!\" id: \"d62d2980-27c4-11ec-92b0-f7b47106bb35\" rule id: \"214ccef6-e98e-493a-98c5-5bcc2d497b79\" signals index: \".siem-signals-spong-default\"", "lastSuccessMessage": "succeeded", "gap": "6 minutes", "lastLookBackDate": "2021-10-07T23:43:27.961Z" }, "type": "siem-detection-engine-rule-status", "references": [ { "id": "d62d2980-27c4-11ec-92b0-f7b47106bb35", <-- previous alertId has been converted to references[] "type": "alert", "name": "alert_0" } ], "migrationVersion": { "siem-detection-engine-rule-status": "7.16.0" }, "coreMigrationVersion": "8.0.0", "updated_at": "2021-10-12T01:50:53.406Z" } }, ``` #### Manual testing --- There are e2e tests but for any manual testing or verification you can do the following: ##### Manual upgrade test If you have a 7.15.0 system and can migrate it forward that is the most straight forward way to ensure this does migrate correctly. You should see that the `Rule Monitoring` table and Rule Details `Failure History` table continue to function without error. ##### Downgrade via script and test migration on kibana reboot If you have a migrated `Rule Status SO` and want to test the migration, you can run the below script to downgrade the status SO then restart Kibana and observe the migration on startup. Note: Since this PR removes the mapping, you would need to [update the SO mapping](https://github.com/elastic/kibana/pull/114585/files#r729386126) to include `alertId` again else you will receive a strict/dynamic mapping error. ```json # Replace id w/ correct Rule Status SO id of existing migrated object POST .kibana/_update/siem-detection-engine-rule-status:d580ca91-2afe-11ec-8621-8d6bfcdfd75e { "script" : { "source": """ ctx._source.migrationVersion['siem-detection-engine-rule-status'] = "7.15.0"; ctx._source['siem-detection-engine-rule-status'].alertId = ctx._source.references[0].id; ctx._source.references.remove(0); """, "lang": "painless" } } ``` Restart Kibana and now it should be migrated correctly and you shouldn't see any errors in your console. You should also see that the `Rule Monitoring` table and Rule Details `Failure History` table continue to function without error. ### Checklist Delete any items that are not applicable to this PR. - [ ] ~[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials~ - [X] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### For maintainers - [x] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) Co-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Garrett Spong <spong@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

…le-status Saved Object migration to SO references (#115355) **Ticket:** #107068 **Follow-up after:** #114585 ## Summary The existing migration function `legacyMigrateRuleAlertIdSOReferences` that migrates `alertId` fields to SO references array did not include all the other attributes of a `siem-detection-engine-rule-status` doc being migrated to the resulting doc. This PR includes a fix and an integration test for that. ## Run the test To run the test, in one terminal execute: ``` cd ${KIBANA_HOME} && node scripts/functional_tests_server --config x-pack/test/detection_engine_api_integration/security_and_spaces/config.ts ``` In another terminal execute: ``` cd ${KIBANA_HOME} && node scripts/functional_test_runner --config x-pack/test/detection_engine_api_integration/security_and_spaces/config.ts --include=x-pack/test/detection_engine_api_integration/security_and_spaces/tests/migrations.ts ``` ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

…le-status Saved Object migration to SO references (elastic#115355) **Ticket:** elastic#107068 **Follow-up after:** elastic#114585 ## Summary The existing migration function `legacyMigrateRuleAlertIdSOReferences` that migrates `alertId` fields to SO references array did not include all the other attributes of a `siem-detection-engine-rule-status` doc being migrated to the resulting doc. This PR includes a fix and an integration test for that. ## Run the test To run the test, in one terminal execute: ``` cd ${KIBANA_HOME} && node scripts/functional_tests_server --config x-pack/test/detection_engine_api_integration/security_and_spaces/config.ts ``` In another terminal execute: ``` cd ${KIBANA_HOME} && node scripts/functional_test_runner --config x-pack/test/detection_engine_api_integration/security_and_spaces/config.ts --include=x-pack/test/detection_engine_api_integration/security_and_spaces/tests/migrations.ts ``` ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

…le-status Saved Object migration to SO references (#115355) (#115486) **Ticket:** #107068 **Follow-up after:** #114585 ## Summary The existing migration function `legacyMigrateRuleAlertIdSOReferences` that migrates `alertId` fields to SO references array did not include all the other attributes of a `siem-detection-engine-rule-status` doc being migrated to the resulting doc. This PR includes a fix and an integration test for that. ## Run the test To run the test, in one terminal execute: ``` cd ${KIBANA_HOME} && node scripts/functional_tests_server --config x-pack/test/detection_engine_api_integration/security_and_spaces/config.ts ``` In another terminal execute: ``` cd ${KIBANA_HOME} && node scripts/functional_test_runner --config x-pack/test/detection_engine_api_integration/security_and_spaces/config.ts --include=x-pack/test/detection_engine_api_integration/security_and_spaces/tests/migrations.ts ``` ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Georgii Gorbachev <georgii.gorbachev@elastic.co>

Adds migration for status SO

bec9779

spong self-assigned this Oct 12, 2021

spong added 4 commits October 12, 2021 10:15

Merge branch 'master' of github.com:elastic/kibana into migrate-siem-…

bbb7fc3

…detection-engine-rule-status-references

Updates namings & deprecations, enabled migration, and added support …

bd30df7

…to references for create API

Merge branch 'master' of github.com:elastic/kibana into migrate-siem-…

0ec7dd6

…detection-engine-rule-status-references

Updates so client findBulk query and removes alertId

122f015

spong marked this pull request as ready for review October 14, 2021 02:13

spong requested a review from a team as a code owner October 14, 2021 02:13

spong added 4 commits October 14, 2021 11:18

Merge branch 'master' of github.com:elastic/kibana into migrate-siem-…

a55c1d0

…detection-engine-rule-status-references

Fixing migration functional test

2bf2bd9

Adds support for reverse_nested to SO aggs and fixes update reference…

503e55b

…s bugs

Merge branch 'master' of github.com:elastic/kibana into migrate-siem-…

e31c9da

…detection-engine-rule-status-references

spong commented Oct 14, 2021

View reviewed changes

Making create/update options param mandatory to ensure correct docume…

5f41332

…nt is always updated/created

spong commented Oct 14, 2021

View reviewed changes

spong requested a review from a team as a code owner October 14, 2021 22:42

Increases happiness of test input parameters

ddccdfe

spong requested review from banderror and xcrzx October 14, 2021 23:11

spong commented Oct 14, 2021

View reviewed changes

...etection_engine/rule_execution_log/saved_objects_adapter/rule_status_saved_objects_client.ts Show resolved Hide resolved

pgayvallet approved these changes Oct 15, 2021

View reviewed changes

...etection_engine/rule_execution_log/saved_objects_adapter/rule_status_saved_objects_client.ts Show resolved Hide resolved

x-pack/plugins/security_solution/server/lib/detection_engine/routes/utils.test.ts Show resolved Hide resolved

rudolf approved these changes Oct 15, 2021

View reviewed changes

Merge branch 'master' into migrate-siem-detection-engine-rule-status-…

8cd206e

…references

Merge branch 'master' into migrate-siem-detection-engine-rule-status-…

248b197

…references

banderror reviewed Oct 18, 2021

View reviewed changes

banderror added the auto-backport Deprecated - use backport:version if exact versions are needed label Oct 18, 2021

banderror approved these changes Oct 18, 2021

View reviewed changes

banderror merged commit fe979e4 into elastic:master Oct 18, 2021

banderror mentioned this pull request Oct 20, 2021

[Security Solution][Detections] Migrate to using Saved Object References array to reference alert saved objects from security saved objects #107068

Closed

spong mentioned this pull request Oct 22, 2021

[SavedObjects] Adds FTR test for identifying failed validation when supporting attribute re-writing for sort #116028

Closed

1 task

rudolf mentioned this pull request Oct 27, 2021

siem-detection-engine-rule-status migration could cause failed upgrade #116423

Closed

banderror mentioned this pull request Nov 11, 2021

[Security Solution] Rule Execution Log - technical debt #118324

Open

19 tasks

spong deleted the migrate-siem-detection-engine-rule-status-references branch April 12, 2022 21:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution] Migrates siem-detection-engine-rule-status alertId to saved object references array #114585

[Security Solution] Migrates siem-detection-engine-rule-status alertId to saved object references array #114585

spong commented Oct 12, 2021 •

edited

Loading

elasticmachine commented Oct 14, 2021

elasticmachine commented Oct 14, 2021

spong Oct 14, 2021

spong Oct 14, 2021

spong Oct 14, 2021

spong Oct 14, 2021

spong Oct 14, 2021

pgayvallet left a comment

banderror commented Oct 18, 2021

banderror commented Oct 18, 2021

banderror left a comment

banderror Oct 18, 2021

spong Oct 19, 2021 •

edited

Loading

banderror Oct 18, 2021

spong Oct 19, 2021

banderror Oct 18, 2021

banderror left a comment

kibanamachine commented Oct 18, 2021

banderror commented Oct 18, 2021

banderror commented Oct 18, 2021

		@@ -111,7 +111,6 @@ export type RuleAlertType = SanitizedAlert<RuleParams>;

		// eslint-disable-next-line @typescript-eslint/no-explicit-any
		export interface IRuleStatusSOAttributes extends Record<string, any> {

		// early return if alertId is not a string as expected
		return { ...doc, references: existingReferences };

[Security Solution] Migrates siem-detection-engine-rule-status alertId to saved object references array #114585

[Security Solution] Migrates siem-detection-engine-rule-status alertId to saved object references array #114585

Conversation

spong commented Oct 12, 2021 • edited Loading

Summary

Manual testing

Manual upgrade test

Downgrade via script and test migration on kibana reboot

Checklist

For maintainers

elasticmachine commented Oct 14, 2021

elasticmachine commented Oct 14, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pgayvallet left a comment

Choose a reason for hiding this comment

banderror commented Oct 18, 2021

banderror commented Oct 18, 2021

banderror left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spong Oct 19, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

banderror left a comment

Choose a reason for hiding this comment

kibanamachine commented Oct 18, 2021

💚 Build Succeeded

Metrics [docs]

Saved Objects .kibana field count

History

banderror commented Oct 18, 2021

banderror commented Oct 18, 2021

spong commented Oct 12, 2021 •

edited

Loading

spong Oct 19, 2021 •

edited

Loading